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The invention relates to image processing and in particular to image 
processing mat involves segmentation of an image into regions of pixel locations with 
corresponding image properties. 

Image segmentation involves grouping of pixel locations into variably 
selectable subsets of connected pixel locations, called segments, for which the pixel values 
have related properties. Ideally, each segment corresponds to a set of pixels where one object, 
or a visually distinguishable part of an object, is visible in the image. Image segmentation can 
be used for various puiposes. In image compression apparatuses, for example, segmentation 
can be used to identify different regions of pixel locations whose content will be encoded at 
least partly by common information such as a common motion vector. As another example, 
in an apparatus that constructs an image of a scene from a user selectable viewpoint on the' 
basis images from different viewpoints, image segmentation can be used to find candidate 
pixel regions that image the same object or background in different images. 

Conventionally, two types of segmentation are known: edge based 
segmentation and core based segmentation. In edge based segmentation segments are defined 
by edges between segments after detecting the edges from the an image. Edges are detected 
for example by taking the Laplacian of image intensity (the sum of the second order 
derivative of the intensity with respect to x position and the second order derivative of the 
intensity with respect to y position) and designating pixel locations where mis derivative 
exceeds a threshold value as edge locations. Subsequently a region surrounded by these edge 
locations is identified as a segment 

Core based segmentation conventionally involves comparing pixel values (or 
quantities computed from pixel values) at each pixel location with a threshold that 
distinguishes between in and out of segment values. Thus, for example, pixels in light regions 
25 of an image can be distinguished from a dark background. 

In both cases the threshold has to be selected on the basis of a compromise. 
Setting the threshold too low makes segmentation susceptible to noise, so that segments are 
identified that do not persist from one image to another, because they do not correspond to 
real objects. Setting the threshold too high may have the effect of missing objects altogether. 
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As aresult foe prior arthas sought for ways of selecting thresholds values that 
on one hand suppress noise effects and on the other hand do not make objects invisible 
Threshold values have been selected adaptive*, on the basis of statistical information about 
the observed pixel values in the image, to achieve optimal distinctions for a give image For 
example, thresholds have been selected on the basis of histograms of the frequency of " 
occurrence of pixel values in the image, between peaks fo foe histogram that are assumed to 
be due to objects and background respectively. Other tecluuques include using median values 
as threshold. 

for individual images, or even as a function of position in individual images represent a 
considerable overhead compared to the basic thresholding operation. 

Nevertheless threshold selection remains a source of error, because it ignores 
coherencebetweenpixel values. Conventional techniques have sought to compensate for this 
by including a -growing" step after thresholding, in whichpixel locations adjacent to 
locations thathave been grouped intoasegment are joined to that segment. As a result foe 
segment depends on foe sequence fo which foe pixels are processed. Au object in foe image 
may be missed altogether if an insufficient number of its pixel locations is identified as 
belongmg to foe same segment As a resmt foreshold errors foat appear to be smaU for pixels 
mdmdually can accumulate to a large error mat misses an object altogether. 

Among others, it is an object of foe invention to provide for a core based 
unage segmentation technique that leads to reliable segmentation results but does not require 
variable threshold selection. 

The invention provides for a method according to Claim 1 . According to foe 
invention foe sign of curvature values of an image intensity at a pixel location is used to 
.dentify foe type of segment to which foe pixel location belongs. Although image intensities 
only assume nonnegative values, foe curvature of their dependence on position can assume 
bofopositive and negative values. As a result a fixed threshold value of zero curvature can be 
used to distinguish regions. Curvature is defined by foe eigenvalues of a matrix of second 
order partial derivatives of foe image intensity as a function of pixel location, but foe 
eigenvalues need not always be computed explicitly to determine foe signs. 

The signs of curvature of foe luminance as a function of pixel location may be 
used for example, but other intensities, such as intensities ofcolor components may be used 
instead or in combination. 
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In an embodiment a pixel location is assigned to different types of region 
according to whether me curvature values at the pixel location are both positive or both 
negative. This provides a robust way of segmenting. In a further embodiment a combination 
of signs of curvature of a plurality of different intensities (for example intensities of different 
color components) is used to assign pixel locations to segments. Thus, more than two 
different types of segment can be distinguished. 

In an embodiment the intensity is low pass filtered and the sign of the 
curvatures is determined after filtering. In this way the effect of noise can be reduced without 
having to select an intensity threshold. The differentiation involved in curvature 
determination is preferably an inherent part of filtering. In a further embodiment the 
bandwidth is set adaptive to image content, for example so as to regulate the number of 
separate regions, or the size (for example the average size) of the regions. 

In another embodiment segments mat are initially determined by assigning 
pixel locations to segments on the basis of sign of curvature are subsequently grown. 
Growing is preferably conditioned on the amplitude of the curvature, for example by joining 
pixel locations with small positive or negative curvature to adjacent segments on condition 
mat the absolute value of the curvature is below a threshold, or by stopping growing when 
the absolute value is above a threshold. 



These and other objects and advantageous aspects of the invention will be 
described using the following figures. 

Figure 1 shows an image processing system 



Figure 1 shows an image processing system that contains an image source 10 
(for example a camera) and an image processing apparatus 1 1, with a first image memory 12, 
a Plurality of filter units 14a-c, a second image memory 16, a segmentation unit 18 and a 
processing unit 19. Image source 10 has an output coupled to first image memory 12, which 
is coupled to the filter units 14a-d. Filter unite 14a-d have outputs coupled to segmentation 
unit 18. Segmentation unit 18 is coupled to second image memory 16. Processing unit 19 is 
coupled to first image memory 12 and to segmentation unit 18 via second image memory 16. 
In operation, image source 10 captures an image and forms an image signal that represents 
intensity I(x,y) of the captured image as a function of pixel location (x,y). The image i 



an 
is 



PHNL030666EPP 



10 



20 



in 



4 02.06.2003 
stored in first memory 12. Segmentation unit 18 identifies groups of pixel locations in the 
image as segments and stores information that identifies the pixel locations in the segments 
second memory 16. Image processing unit 19 uses the information about the segments in the 
image to process the image, for example during a computation of compressed image signal 
for storage or transmission purposes or to construct a displayable image signal from a 
combination of images from image source 10. 

Filter units 14a-c each perform a combination of low pass filtering of the 
intensity I(x,y) and taking a second order derivative of toe low pass filtered version of the 
intensity I(x,y). Each filter unit 14a-c determines a different second order derivative from the 
set that includes the second derivative with respect to position along an x direction, the 
second derivative with respect to position along a y-direction and a cross derivative with 
respect to position along the x and y direction. Expressed in terms of a basic filter kernel 
G(x,y) the filter kernels of the respective filter units 14a-c are defined by 

15 Gx*(x J y)=5 2 G(x,y)/dx 2 
Gjyfry^GCjcyyay 2 
G^y^S 2 G(x,y)/ SxSy 



Filter units 14a,c compute images Lot, Iyy, Ley corresponding to 

I** (x,y) = J dx'dy* G^x-x'.y-yO KAtf 
Iyy (x,y) = J dx'dy* GyyCx-x'.y-y') I(x',y>) 
W (*,y) - J dx'dy- G^x-x'.y-yO I^y*) 

25 For the sake of clarity these filter operations have been formulated in terms of integrals, 

although of course the intensity is usually sampled at discrete pixel locations (x,y). Therefore, 
filter units 14a-c normally compute sums corresponding to the integrals. 
The derivative filtered images I^y), I„ (x,y) and 1^ (x,y) define a matrix 

30 Mx,y) Ixy(x,y) 

x xy(x,y) Iyy(x,y) 



For each pixel location x,y. The eigenvalues of this matrix define the curvature of the 
intensity I(x,y) at the location (x,y) after filtering. 
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Segmentation unit 18 uses a combination of the signs of the eigenvalues to segment me 
image. In one embodiment pixel locations where both eigenvalues are positive are assigned 
to segments of a first type and pixel locations where both eigenvalues are negative are 
assigned to segments of a second type. It is not necessary to compute the eigenvalues 
5 explicitly to determine the signs. The determinant of the matrix 



10 



DCx.yHxxCx.y^yyCx^-lVx^) 
equals the product of the eigenvalues. The trace 
T(x,y)=I„(x,y) + I jy (x J y) 



are 



equals the sum of the eigenvalues. Hence it can be determined that both eigenvalues i 
positive at a pixel location by detecting that both the detenninant and the trace for that pixel 

15 location are positive and and it can be detected that both eigenvalues are negative for a pixel 
location when the determinant is positive and the trace is negative for that location. 
Segmentation unit 18 initially deteimines for each individual pixel locations whether it 
belongs to a first type of segment, to a second type of segment or to neither of these types. 
Next, segmentation unit forms groups of pixels locations that are neigbors of one another and 

20 belong to the same type of segment Each group corresponds to a segment Segmentation unit 
18 signals to processing unit 19 which pixel locations belong to the same segment This may 
be done for example by using different image mapped memory locations in second memory 
16 for different pixels location and writing label values that identify different regions to 
which the pixel locations belong into the different locations. In another embodiment 

25 segmentation unit does not identify the regions individually, but only writes information into 
memory locations to identifies the type of region to which the associated pixel location 
belongs. It should be appreciated that instead of storing information for all pixel locations 
information may be stored for a subsampled subset of pixel locations, or in a non memory 
mapped form, such as boundary descriptions of different segments. 

30 Processing unit 19 uses the information about the segments. The invention is 

not limited to a particular use. As an example processing unit 19 may use segments of the 
same type that have been found in different images in a search for corresponding regions in 
different images. When a first segment occurs in a first image and a second segment of the 
same type occurs in a second image processing unit 19 checks whether the content of the first 
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and second images matches in or around the segments. If so, this can be used to compress lie 
images, by coding the matching region in one image with a motion vector relative to the 
other. The motion vectors may be applied to encoding using the MPEG standard for example 
(the MPEG standard is silent on how motion vectors should be determined). An alternative 
use could be the determination of the distance of an object to the camera from the amount of 
movement. The segmente may also be used for image recognition purposes. In an 
embodiment segmente of one type only are selected for matching, but in another embodiment 
all types of segment are used. 

Processing efficiency of processing unit 1 9 is considerably increased by using 
segments with similar curvature to select regions for determining whether the image content 
matches and by avoiding such selection if there are no segmente with curvature does not 
match. The sign of the curvature is a robust parameter for selecting segmente, because it is 
invariant under many image deformations, such as rotations, translations etc. Also, many 
gradual changes of mumination leave the signs of curvature invariant, since the signs of 
curvature of in an image region that images an object are strongly dependent on toe intrinsic 
three-dimensional shape of the object 

Although the operation of segmentation unit 18 has been described in terms of 
a one to one relation between detected sign of the curvatures and assignment to a segment 
However, without deviating from the invention segmentation unit 18 may apply a growing 
operation to determine the segment, joining pixel locations that are adjacent to a segment but 
have not been assigned to the segment to that segment and merging segmente that become 
adjacent in this way. Growing may be repeated iteratively until segmente of opposite type 
meet one another. In an alternative embodiment growing may be repeated until the segmente 
reach pixel locations where edges have been detected in the image. 

Growing segmente is known per se, but according to the invention the sign of 
the curvatures is used to make an initial segment selection. An implementation of growing 
involves first writing initial segment type identifications into image mapped memory 
locations in second memory 16 according to the sign of curvature for the pixel locations and 
subsequently changing these type identifications according to the growing operation for 
example by writing the type identification of pixel locations of an adjacent segment into a 
memory location for apixel location that is joined to that segment 

The opposite of growing, shrinking, maybe used as well, for example to 
remove irregularities on the edges of the selected segmente. 
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In an embodiment segmentation unit conditions growing on the amplitude of 
the detected curvatures. In this embodiment pixel locations for which the curvatures have a 
sign opposite to that of an adjacent segment are joined to that segment when the one or more 
of the amplitudes of the curvatures for the pixel location are below a first threshold and one 
5 or more of the curvatures for the segment are above a second threshold. The thresholds may 
have predetermined values, or may be selected relative to one another. 

As described segmentation unit 18 preferably distinguishes two types of initial 
segment, with pixel locations that have positive-positive or negative-negative curvature 
values respectively. However, different types of segment types may be used, for example 

10 with pixel locations where the in absolute sense largest curvature values are positive and 
negative respectively 

Furthermore, curvature of luminance information as a function of pixel 
location is preferably used to select the regions, in other embodiments one may of course also 
use the intensity of other image components, such as color components R, G or B or U or V, 

15 or combinations thereof (the R, G, B, U and V components have standard definitions). In a 
further embodiment curvatures are determined for a plurality of different components and the 
combination of the signs of the curvatures for different components is used to segment the 
image. Thus, more than two different types of segments may be distinguished, or different 
criteria for selecting segments may be used. For example, in case of curvatures of R, G and 

20 B, three pieces of sign information may be computed, encoding for the R, G and B 

component respectively, whether the curvatures of the relevant component ace both positive, 
both negative or otherwise. These three pieces of sign information may be used to distinguish 
eight types of segments (R, G and B curvatures all both positive, R and G curvatures all both 
positive and B curvatures both negative, etc.). These eight types may be used to segment the 

25 image into eight types of segments. Thus a more selective preselection of regions for 
matching by processing unit 19 is made possible. 

A smaller number of types may also be used, for example a first type where at 
least two of the R, G and B components have all positive curvatures and a second type where 
at least two of the R, G and B components have all negative curvatures 

30 In a preferred embodiment filter units 14a-c use a Gaussian kernel G(x,y). 
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This type of Kernel has the advantage that it can be implemented in filter units 
14a-c as a cascade of two one-dimensional filter operations. 

In an embodiment the filter scale (a in case of Gaussian filters) is selected 
adaptive to image content. In one example segmentation unit 18 compares the number of 
initially determined regions with a target number and increases or decreases the scale when 
the number of initially determined regions is above or below the target number respectively 
Instead of a single target value a pair of target values may be used, the scale being increased 
when the number of initially determined regions is above an upper threshold and decreased 
when that number is below a lower threshold. Thus, noise effects can be reduced without 
having to select an intensity threshold. As an alternative, the average size of the regions may 
be used instead of the number of regions to control adaptation of the scale. 

The various units shown in figure 1 may be implemented for example in a 
suitably programmed computer, or digital signal processor unit that is programmed or 
hardwired to perform the required operations, such as filtering, sign of curvature 
compulation, initial assignment to segments on the basis of the signs of curvature and 
segment growing. Instead of programmable processors dedicated processing units may be 
used, which may process the image intensity as a digital or analog signal or a combination of 
both. Combinations of these different types of hardware may be used as well. 
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CLAIMS: 



1 • An image processing method comprising segmentation of an image, said 

segmentation comprising the steps of 

computing, for respective pixel locations in the image, information about signs 
of curvature values of an intensity of the image as a function of pixel location; 
5 " assigning pixel locations to different segments, each according to one or more, 

or a combination of the signs for the pixel location. 

2. An image processing method according to Claim 1, comprising 

assigning each pixel location to respective different type of segments 
10 according to whether the signs of the curvature values in two mutually transverse directions 
at the pixel location are both positive or both negative respectively. 

3- An image processing method according to Claim 1, comprising spatially low 
pass filtering the intensity prior to said computing and computing the information about the 

15 sign of curvature from the low pass filtered intensity. 

4- An image processing method according to Claim 3, comprising selecting a 
bandwidth of said low pass filtering adaptive to a content of the image. 

20 5. An image processing method according to Claim 1 , comprising growing the 

segments initially determined by said assigning, wherein said growing is conditioned on an 
amplitude of the curvature values. 

6. An image processing apparatus, comprising 

25 a sign of curvature computation unit (14a-c, 1 8) arranged to compute, for 

respective pixel locations, information about signs of curvature values of an intensity of the 
image as a function of pixel location; 

a segmentation unit (18), arranged to assign pixel locations to different 
segments each according to one or more, or a combination of the signs for the pixel location. 
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7. An image processing apparatus according to Claim 6, wherein the 
segmentation unit (18) is arranged to assign each pixel location to respective diflerent types 
of segment when the signs of the curvature values in two mutually transverse directions at the 
pixel location are hoth positive or hoth negative respectively. 

8. An image processing apparatus according to Claim 6, comprising a spatial low 
pass filter unit (14a-c), for filtering the intensity prior to said computation of the information 
about the sign. 

9. An image processing apparatus according to Claim 8, comprising a feedback 
loop for selecting a bandwidth of said low pass filtering adaptive to a count of selected 



10. An image processing apparatus according to Claim 8, comprising a feedback 

loop for selecting a bandwidth of said low pass filtering adaptive to a size of selected 



11. An image processing apparatus according to Claim 6, wherein the 

segmentation unit is arranged to grow the segments initially determined by said 
wherein said growing is conditioned on an amplitude of the curvature values. 
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ABSTRACT: 



A image obtained by a camera is segmented into regions. Information about 
signs of curvature values of an intensity of the image is computed as a function of pixel 
location. Pixel locations are assigned to different segments, each according to one or more, or 
a combination of the signs for the pixel location. Preferably, each pixel location is assigned to 
5 a respective type of segment according to whether the signs of the curvature values in two 
mutually transverse directions at the pixel location are both positive or both negative 
respectively. Preferably spatial low pass filtering is used to control the number of segments 
that are found in this way. 



10 Fig. 1 
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